{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7614b2de",
   "metadata": {},
   "source": [
    "# Example: Train a Recommender\n",
    "\n",
    "In this example, we want to guide you through the process of training a recommender from offline evaluations. You can then use this recommender to provide you with good default choices for models and hyperparameters on new datasets. All without ever training a model on that dataset!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1351d0dd",
   "metadata": {},
   "source": [
    "## Initialize the Tracker\n",
    "\n",
    "As usually, we first need to initialize a tracker to access the offline evaluations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6d4f99db",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from tsbench.evaluations.tracking import ModelTracker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "99f1bfe6",
   "metadata": {},
   "outputs": [],
   "source": [
    "tracker = ModelTracker.from_directory(Path.home() / \"evaluations\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92cb1c79",
   "metadata": {},
   "source": [
    "## Initialize the Surrogate\n",
    "\n",
    "Using the tracker, we can easily train a surrogate model which learns from all performances available from the tracker. Here, we use a ranking MLP as it provides the best performance.\n",
    "\n",
    "Further, we want the surrogate model to predict an accuracy metric (the nCRPS) and the latency. Thus, we can later obtain models from the recommender which are both accurate and fast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a4580d0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tsbench.surrogate import MLPSurrogate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c2b8d449",
   "metadata": {},
   "outputs": [],
   "source": [
    "surrogate = MLPSurrogate(\n",
    "    tracker,\n",
    "    objective=\"ranking\",\n",
    "    discount=\"linear\",\n",
    "    hidden_layer_sizes=[32, 32],\n",
    "    predict=[\"ncrps_mean\", \"latency_mean\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36fd5625",
   "metadata": {},
   "source": [
    "## Train the Recommender\n",
    "\n",
    "The surrogate can subsequently be passed to a recommender which is responsible for the multi-objective selection. We focus the selection on the nCRPS such that the first recommended model optimizes for nCRPS, the second one for latency, and the third one provides a tradeoff."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c3e4251a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tsbench.recommender import ParetoRecommender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1256fb79",
   "metadata": {},
   "outputs": [],
   "source": [
    "recommender = ParetoRecommender(\n",
    "    surrogate, objectives=[\"ncrps_mean\", \"latency_mean\"], focus=\"ncrps_mean\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3069edc1",
   "metadata": {},
   "source": [
    "Having initialized the recommender, we can fit it. We will use all available data since we won't evaluate the performance on a test set here. Note that, since we are using the MLP surrogate, training takes a few dozen seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c556a0f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = tracker.get_evaluations()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "20d68c8c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: False, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "/home/ubuntu/tsbench/.venv/lib/python3.8/site-packages/pytorch_lightning/trainer/data_loading.py:132: UserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 48 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  rank_zero_warn(\n",
      "GPU available: False, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n"
     ]
    }
   ],
   "source": [
    "recommender.fit(data.configurations, data.performances)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "763c6abb",
   "metadata": {},
   "source": [
    "## Predicting Default Models\n",
    "\n",
    "Once the recommender is trained, we can use it to predict models for unseen datasets. While we are required to pass a dataset here, it will be ignored since the specific surrogate model that we used does not use dataset meta features for its prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "62660dda",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tsbench.config import DATASET_REGISTRY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e99d0683",
   "metadata": {},
   "outputs": [],
   "source": [
    "dummy_dataset = DATASET_REGISTRY[\"m4_monthly\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3cb64a9f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: False, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "/home/ubuntu/tsbench/.venv/lib/python3.8/site-packages/pytorch_lightning/trainer/data_loading.py:132: UserWarning: The dataloader, predict_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 48 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n",
      "  rank_zero_warn(\n",
      "GPU available: False, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n"
     ]
    }
   ],
   "source": [
    "recommendations = recommender.recommend(dummy_dataset, max_count=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd8e7ddd",
   "metadata": {},
   "source": [
    "As we can see, the first recommendation is a large DeepAR model (4 layers and 80 cells) which apparently tends to be very accurate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f9506926",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DeepARModelConfig(training_fraction=1.0, learning_rate=0.001, context_length_multiple=1, num_layers=4, num_cells=80)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommendations[0].config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70c65761",
   "metadata": {},
   "source": [
    "The second recommendation is a Seasonal Naïve model, which is arguably very fast, independently of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7609149d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SeasonalNaiveModelConfig()"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommendations[1].config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5cfc0ee",
   "metadata": {},
   "source": [
    "The third recommendation is a Simple Feedforward model with 3 hidden layers and a non-standard context length multiple of 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "117631d3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SimpleFeedforwardModelConfig(training_fraction=0.3333333333333333, learning_rate=0.001, context_length_multiple=2, hidden_dim=60, num_layers=3)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommendations[2].config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bc14fa1",
   "metadata": {},
   "source": [
    "If you were to find a model for an unseen dataset, you could now train the first few recommendations of the recommender on your data to figure out which one actually performs best (in terms of latency and accuracy)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98b61460",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
